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NEAR REAL-TIME DISSEMINATION OF SURVEILLANCE VIDEO 
Related Applications 

[0001] This application claims the benefit of U.S. Provisional Application No. 

60/447,744 filed February 19, 2003 entitled "Near Real-Time Dissemination of Surveillance 
Video," the disclosure of which is entirely incorporated herein by reference. 

Technical Field 

[0002] The present subject matter relates to systems, techniques and software for the 

smart dissemination of video surveillance information in near real-time, for example, to tactical 
users in the field via limited bandwidth communication facilities. 

Background 

[0003] Demand for bandwidth has continued to grow as the military has developed more 

data-intensive weapons and intelligence surveillance and reconnaissance systems. However, 
scarcity/limitations on both military and commercial communication links and additional 
bandwidth required to transmit streaming video from an unmanned aerial vehicle (UAV) or 
satellite sources has limited dissemination of such information, especially to the lowest echelon 
forces. As the availability of strategic surveillance video expands, the need to effectively 
distribute such information to the forces needing to use it will grow exponentially. It is unlikely 
that the bandwidth to transport such information can keep pace. 

[0004] Presently, only one channel of UAV video requires bandwidth in the megabit per 

second range. Couple this with the enormous bandwidth consumed by the GLOBAL HAWK 
drone, and demand for bandwidth stands as a major barrier to beaming this timely intelligence 
information to US ground, air, and naval units for planning and direct military support purposes. 
In addition, the viewing of current analog collection video is time consuming and labor intensive, 
especially in retrieving past information, which can require lengthy serial searches of data tapes. 
Furthermore, the conventional method of viewing the earth through a narrow field of view sensor 
(equivalent to looking out an airplane window with a soda straw) does not provide a geographic 
context (e.g., "Where am I?", and "What's ahead?"). 
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[0005] Similar problems arise in other operations that could take advantage of timely 

distribution of video information to personnel in the filed, such as distribution of surveillance 
video to police and homeland security operatives. 

[0006] A need exists for an enhanced technique for providing solutions to these 

problems. Specifically, there is a need to provide wide dissemination of tactically useful 
portions of surveillance video to the particular users in the field who need relevant portions of 
the video information. Any such solution should provide near real-time distribution of the 
information yet require minimal bandwidth on the links to the end users. The distribution system 
should offer intelligence to military forces or other users operating round-the-clock, anywhere on 
the globe. Also, the solution should be readily adaptable to virtually any user terminal 
equipment and/or user communication capability available in the field. 

Summary 

[0007] The inventive concepts alleviate one or more of the above noted issues or needs 

regarding dissemination of video intelligence information for UAV sources and other video 
surveillance platforms. The disclosed near real-time video (NRTV) system provides a unique 
solution to the problems outlined above and addresses other issues relating to practical 
dissemination of tactical video information to users in the field of operations. The disclosed 
system encapsulates a smart pull paradigm, intuitive graphics-driven notification, and a unique 
reach back mechanism capable of delivering time-sensitive surveillance video clips and/or still 
images, at a fraction of the bandwidth required by traditional streaming video, for example, to 
military forces operating round-the-clock, anywhere on the globe. 

[0008] Methodologies for the distribution of video surveillance information are 

implemented on a server and/or on a terminal device. For example, a server method for 
disseminating video surveillance information might entail receiving and storing real-time video 
image information from a camera mounted on a moveable surveillance platform, over a period of 
time. With the video image information, the server also receives position and orientation 
telemetry regarding operation of the camera. The server forwards information regarding the 
telemetry, in notification messages sent to a user's client device. When the server receives an 
information request from the user's client device, which identifies a user selected portion of the 
received video image information, the server transmits the selected video image information to 
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the user's client device, for example, for display as a short video clip or for display of selected 
frames. 

[0009] As another example, a terminal or client device oriented method, for obtaining 

and presenting video surveillance information, involves receiving position and orientation 
telemetry regarding operation of a camera mounted on a moveable surveillance platform, over a 
period of time. The terminal processes the telemetry, to determine a view of the camera over the 
period of time; and it displays the determined view of the camera over time, on a user interface. 
The terminal detects a user selection of a view via the user interface. The selected view 
corresponds to a portion of the period of time. In response, the terminal formulates an 
information request indicating the corresponding portion of the time period and sends that 
request to* a facility having access to stored video information received from the camera. The 
terminal receives one or more frames of video information generated by the camera, from the 
facility. The one or more frames contain actual image data taken by the camera during the 
corresponding portion of the period of time and thus correspond to the user selected view; and 
the terminal displays the received video information to the user. 

[0010] As disclosed, a typical NRTV system consists of a server component and a client 

component. The server component consists of a set of processing equipment and software that 
can be rack mounted or mounted in a movable transit case. The server component interfaces to 
one or more user communications components, depending upon user requirements. The client 
component also interfaces to one or more user communications components and consists of a 
processing system (typically a PC - desktop or laptop) and an information management/display 
software package. 

[0011] The server component provides the capability to input unencrypted video in any 

analog or digital video format (NTSC, PAL, MPEG, etc.) from manned or unmanned video 
surveillance systems, such as the Predator UAV, digitize (encode and timestamp), capture frames 
at specified intervals (or based upon various trigger events), encode and time stamp the frames, 
and store all data locally for an entire collection mission (up to about 8 hours). In near real time 
it also can process (parse, error correct, and store locally as an ASCII file) metadata about the 
sensor collection parametrics (telemetry) that are generated at prescribed intervals and 
distributed in the data stream from the sensor, in parallel with the video processing. This 
functionality is made possible through the integration of commercially available hardware and 
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software. The server additionally provides the ability to manage the transmission of data files 
between the server and the client in a synchronous or asynchronous manner to accommodate a 
range of network or radio communication interfaces to/from a user. The video processor 
software also provides the capability to add literal annotations to a video frame. The annotations 
are then included in the video availability notice (VAN) message and passed to the user by the 
communications processor. The annotations are then displayed by the client software to provide 
the user with content information about the video. 

[0012] The client component can receive and process the sensor collection metadata and 

generate and display graphic overlays that represent the outline of the video frames and thus the 
field of view of the video camera for each of the frames. The client displays these frame outline 
overlays together with content annotations, if any, that were captured by the server component at 
prescribed intervals as well as the location of the collection platform, all in near real time relative 
to the actual collection of the video data by the collection platform. The communications with 
the client device use available user communications resources. The client device provides 
mapping software that allows the overlays to be displayed over digital map data, to provide full 
geographic context to the sensor collection field of view. Thus the user is allowed to visually 
become aware of the remote sensor collection in near real time. The client component also 
allows the user to graphically request a video clip from what has been collected up to the point in 
time selected for the on-going mission and handle the transmission of that request to the server 
component over user available communications. The client also can display the resulting video 
clip to the user, when it arrives over the user communications system. 

[0013] Aspects of the NRTV concept relate to systems and computer components, such 

as the server and the client device, programmed to implement the unique functions of the NRTV 
system outlined above. Other concepts relate to unique software for implementing the server and 
client functions on programmable computer devices. A software product, in accord with this 
concept, includes information that may be carried by at least one machine-readable medium and 
information. The information carried by the medium may be executable code, one or more 
databases and/or information regarding surveillance video available through the NRTV 
dissemination technology. 

[0014] A computer or machine "readable medium, 55 as used herein, may be any physical 

element or carrier wave, which can bear instructions or code for performing a sequence of steps 
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in a machine-readable form or associated data. Examples of physical forms of such media 
include floppy disks, flexible disks, hard disks, magnetic tape, any other magnetic medium, a 
CD-ROM, any other optical medium, a RAM, a ROM, a PROM, an EPROM, a FLASH- 
EPROM, any other memory chip or cartridge, as well as media bearing the software in a 
scannable format. A carrier wave type of medium is any type of signal that may carry digital 
information representative of the data or the instructions or code for performing the sequence of 
steps. Such a carrier wave may be received via a wireline or fiber-optic network, via a modem, 
or as a radio-frequency or infrared signal, or any other type of signal which a computer or the 
like may receive and decode. 

[0015] Additional advantages and novel features of the examples will be set forth in part 

in the description which follows, and in part will become apparent to those skilled in the art upon 
examination of the following and the accompanying drawings or may be learned by production 
or operation of the examples. The advantages of the present subject matter may be realized and 
attained by means of the methodologies, instrumentalities and combinations particularly pointed 
out in the appended claims. 

Brief Description of the Drawings 

[0016] The drawing figures depict one or more implementations in accord with the 

present concepts, by way of example only, not by way of limitations. In the figures, like 
reference numerals refer to the same or similar elements. 

[0017] Fig. 1 a high-level functional block diagram of a video surveillance system with 

near real-time dissemination of portions of the video. 

[0018] Fig. 2 is a simplified diagram of the video components on board a surveillance 

platform. 

[0019] Fig. 3 is a simplified block diagram of a general purpose computer, which may 

serve as the client device in the system/network of Fig. 1 . 

[0020] Fig. 4 is a simplified block diagram of a general purpose computer, which may 

operate as the server device in the system/network of Fig. 1. 

[0021] Figs. 5 to 7 are a series of flow diagrams, useful in explaining the 

communications flow during various stages of operation of the system, including notification, 
reachback and dissemination. 
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[0022] Figs. 8 to 11, show screen shots of each stage in the process of notifying, 

requesting and receiving video related information, as presented by the client software 
application. 

[0023] Fig. 12 illustrates the field of view geometry and is helpful in understanding the 

determination of the filed of view overylays. 

Detailed Description 

[0024] The various examples disclosed herein relate to systems, techniques and software 

for near real-time dissemination of video data from surveillance systems, such as manned and 
unmanned aerial surveillance platforms. The exemplary system processes one video source 
signal from a surveillance platform. If the source signal is analog, the system digitizes the video. 
The system allows a local or remote user to visualize frames representing the video field of view 
at set times, on a standard map display of the area under surveillance. From an overlay of the 
field(s) of view, the system allows the user to reach back and request a video clip from the actual 
video surveillance information. Using available communications, the request is relayed to the 
video processing subsystem, and a video file of the video collected for the requested time period 
is generated and sent back to the user for display, all in near real-time. The communication of 
the request and downloading of the video clip, however, requires far less bandwidth than real- 
time streaming or analog transmission of the entire surveillance video. 

[0025] Reference now is made in detail to the examples illustrated in the accompanying 

drawings and discussed below. Fig. 1 provides a high-level view of the elements of a system 
that might implement the near real-time video dissemination. As shown, the system comprises a 
video data source 1, a server 3 and a client component 5. The server 3 includes a video 
processor system 7 and a communications processor 9. The video processor system 7 may be 
implemented in hardware and/or software of a general purpose computer functioning as the 
server 3. The communication processor provides physical and logical interfacing for 
communication with a number of client devices 5, via one or more available communication 
paths 1 1 . 

[0026] The system of Fig. 1 processes and disseminates video information obtained from 

one or more surveillance platforms. The system may work with video and telemetry from any 
appropriate platform. It is assumed that those skilled in the art are familiar with such platforms 
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and the structure and operations of a wide variety of such platforms. However, to insure 
completeness and facilitate understanding by some readers, a brief summary discussion of 
exemplary video surveillance platforms follows. 

[0027] Fig. 2 provides a simplified block diagram of the video processing elements, as 

they might be used on an aerial video surveillance platform. The platform may be a satellite or a 
manned aircraft, commonly used for military, police, anti-terrorism, or other surveillance 
applications. The platform could be mounted on a land vehicle, or in some cases moveably 
mounted at a relatively fixed location. For purposes of further discussion, however, the example 
will be assumed to use an unmanned aerial vehicle (UAV). 

[0028] As shown in Fig. 2, the video data source 1 includes a camera 15 manipulated by 

a robotic control mechanism 17, for example mounted on a UAV (not shown). The camera 
images objects within its field of view and generates a responsive video signal, in real time. The 
robotic control mechanism 17 controls the horizontal direction of the camera 15 relative to the 
flight path of the UAV as well as the elevation (up-down) angle of the camera 15. The robotic 
control mechanism 17 controls the field of view / of the camera 15. The robotic control 
mechanism 17 is operated by a controller 19 and may provide feedback signals to the controller 
19. In this manner, the controller 19 maintains data regarding the directional orientation and 
elevation of the camera 15 relative to the UAV flight path, as well as data regarding the field of 
view / of the camera 15. 

[0029] A global positioning system (GPS) processor 21 constantly receives and processes 

satellite signals to determine the three-dimensional coordinates of the UAV. The processor 
supplies the position data to the controller 19. The video signal may be an analog video signal 
(NTSC or PAL formats), typically from a UAV such as the Predator UAV. Alternatively, the 
video may be a digital signal, in which case the system will not have to digitize the video signal, 
but will process the digital video format. A video processor 23 receives the video signal from 
the camera 15, the GPS data regarding the position of the UAV and the data regarding camera 
orientation and field of view and multiplexes the various signals and data together for 
transmission. A transmitter 25 sends the combined signal from an antenna 27 on the UAV over a 
wireless link and an appropriate network (not separately shown) to the server 3. 
[0030] Returning to Fig. 1, the video signal (which is assumed to be unencrypted at this 

point) is fed into the video processor sub-component 7, where the telemetry data is separated out, 
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indexed a nd s eparately stored. If t he s ource v ideo s ignal i s a nalog, t he v ideo p rocessor s ub- 
component 7 digitizes the video data. Those skilled in the art will recognize that the video 
components on the UAV may produce a digital video signal. The video processor sub- 
component 7 indexes and separately stores the digitized video signal, for future access. 
[0031] The video processor sub-component 7 also generates a data file of the telemetry 

information that it passes on to the communications processor sub-component 9, for delivery to 
the clients 5. These data files serve as the basis for the Video Availability Notification messages, 
or VANs. Each VAN provides sufficient information to allow a client device to determine the 
corner points, in latitude and longitude, of the camera sensor field of view at a point in time, as 
projected onto a map of the area under surveillance. The camera sensor field of view in relation 
to the ground is calculated from the parsed values of target latitude and target longitude (which 
represent the center of the camera field of view), angular sensor field-of-view /, and sensor 
azimuth. First (See Fig. 12), the target latitude and longitude is used to define an initial Earth 
Centered Fixed (ECF) vector. Next, the sensor field-of-view and azimuth are used to define the 
angles between the field of view center point and each corner point. The ECF vector of the^ 
target location is rotated through each of the respective angles generated above to determine the 
latitude and longitude of each corner point. 

[0032] An NRTV client polygon overlay is then created and the four geographic corner 

coordinates are stored in the overlay. One overlay is generated for each validated VAN. All the 
overlays for each generated frame are stored in a NRTV client UAV overlay. When the UAV 
overlay is painted onto a map background, the mapping software generates device point 
coordinates for each geographic coordinate stored in each of the NRTV polygons. 
[0033] The server periodically g enerates a VAN in response to the telemetry data and 

sends the VAN to each client device that is monitoring the current UAV surveillance mission. 
The client 5 displays one or more of the VANs as rectangular or square geographic overlays on a 
map background, for example, as shown in Fig. 8. In this manner, frames from the incoming 
video are virtually sampled by the server at regular intervals (generating a VAN for one frame 
every 3-4 seconds), and each virtual frame is displayed on the client device as a field of view 
rectangle overlaid on the map. In Fig. 8, the current frame is represented as a rectangle 31, with 
the previous nine frames shown as rectangles 33. Although the data is available as stored in the 
client device, earlier frame overlays are not shown on the display unless the user selects to scroll 
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back through the mission data. The map display may also show a projection of the current 
location of the UAV, for example, as an icon for an airplane 34, based on the latitude and 
longitude data in the most recent VAN message. In an example with a color display, the current 
frame rectangle 31 might be shown in red, whereas the rectangles 33 for the previous nine 
frames might appear in blue. As each new VAN is processed to display a new frame, the 
preceding frame will be changed from red to blue, with a total of ten frames always displayed. 
The UAV icon 34 may also appear in red. 

[0034] The user then has the option of requesting a video clip, or one or more still 

frames, from the video collection information. The user begins the process by selecting the 
UAV frame information (a left mouse click over one of the displayed frames) and then making a 
right mouse click. This will cause the client device to display the option to select a video clip, as 
shown at 35 in Fig. 9. After the user clicks on Select Video Clip, a new dialog box 37 (Fig. 10) 
will appear with a slider bar 39, to select the start and end times of the video clip. 
[0035] Fig. 10 shows the method by which a user would identify the start and end times 

of the desired video clip based upon the frame information, and then request that clip. The user 
begins the process by selecting (left mouse click and hold) the one of the slider bar handles 41, 
43 (the small, solid black rectangles at the ends of a cyan colored part of the bar in the Select 
UAV Playback Times window). When the user slides the left handle 41 to the right, the Start 
Time window moves forward in time and the video frames corresponding to the start and end 
times are selected (change color), to indicate what camera field of view areas will be represented 
in the corresponding video clip. Selecting the right most slider bar handle 43 changes the end 
time in a similar manner. Other selection input techniques may be used. 

[0036] Once the user has adjusted the slider bar 39 to select the start and end times of the 

requested video clip, the start and end frames will be highlighted on the map (for example as 
yellow rectangles). The user can then select the Request Clip button 45. Once this is done the 
information is relayed back to the communications processor 9 in the server 3, and then sent on 
to the video processor 7 to extract the appropriate stills or video clip from the stored information. 
The server 3 compiles a file of the requested/extracted video, essentially forming one or more 
frames of a video clip, and sends the file back to the user client 5 that requested it, as shown in 
Fig. 1 1 . The video clip is displayed, for example, in a separate window 47 using the Microsoft 
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Media Player. The video can be started and stopped at any point in the clip. Annotations can 
also be added to any video frame using the video processing software. 

[0037] The various client devices 5 and the server 3 shown in the drawings are fairly well 

known general purpose computers and/or mobile computing devices. It is assumed that those of 
skill in the relevant arts will be familiar with the structure, programming and operations of such 
devices. However, to insure adequacy of the teaching here to various readers, it may be helpful 
to briefly review the relevant technologies. 

[0038] Fig. 3 is a functional block diagram of a PC or workstation type implementation 

of a system 151, which may serve as one of the client devices 5, for accessing the video 
surveillance data. The exemplary computer system 151 contains a central processing unit (CPU) 
152, memories 153 and an interconnect bus 154. The CPU 152 may contain a single 
microprocessor, or may contain a plurality of microprocessors for configuring the computer 
system 152 as a multi-processor system. The memories 153 include a main memory, a read only 
memory, and mass storage devices such as various disk drives, tape drives, etc. The main 
memory typically includes dynamic random access memory (DRAM) and high-speed cache 
memory. In operation, the main memory stores at least portions of data and of instructions for 
execution by the CPU 152. 

[0039] The mass storage may include one or more magnetic disk or tape drives or optical 

disk drives, for storing data and instructions for use by CPU 152. For a laptop PC, for example, 
at least one mass storage system 155 in the form of a disk drive or tape drive, stores the operating 
system and application software as well as data, including received messages and documents, 
and for purposes of the video application, will store received camera telemetry information and 
received video surveillance data. The mass storage 155 within the computer system 151 may 
also include one or more drives for various portable media, such as a floppy disk, a compact disk 
read only memory (CD-ROM), or an integrated circuit non-volatile memory adapter (i.e. 
PCMCIA adapter), to input and output data and code to and from the computer system 151. 
[0040] The computer system 151 includes appropriate input/output ports 156 for 

interconnection with a display 157 and a keyboard 158 serving as the respective user interface. 
For example, the computer may include a graphics subsystem to drive the output display 157. 
The output display 157 may include a cathode ray tube (CRT) display or liquid crystal display 
(LCD). Although not shown, the PC type system typically would include a port for connection 
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to a p rinter. T he i nput c ontrol d evices f or s uch a n i mplementation o f t he sy stem 1 5 1 w ould 
include the keyboard 158 for inputting alphanumeric and other key information. The input 
control devices for the system may further include a cursor control device (not shown), such as a 
touchpad, a mouse, a trackball, stylus, or cursor direction keys. The links of the peripherals 157, 
158 to the system 151 may be wired connections or use wireless communications. 
[0041] The system 151 also includes one or more input/output interfaces for 

communications, shown by way of example as an interface 159 for data communications via a 
network. The physical communication links may be optical, wired, or wireless (e.g., via satellite 
network, etc.). For some users, the interface 159 may provide landline-based communications, 
e.g. via data communication modem or local area network card or the like. However, for 
personnel at remote locations in the field, the interface 159 provides a wireless communication 
link. The communication link may be broadband, but in many instances, the bandwidth of the 
link is relatively limited. 

[0042] Each computer system 151 runs an operating system as well as a variety of 

applications programs and stores data, enabling one or more interactions via the user interface, 
provided through elements such as 157 and 158, and/or over the network 51 to implement the 
desired processing for the video service application. The client computer 151, for example, runs 
a map program, a media player, etc. to implement the client programming for the NRTV video 
presentation functions described and shown herein. 

[0043] Fig. 4 is a functional block diagram of a general purpose computer system 251, 

which may perform the functions of the server 3. The exemplary computer system 251 contains 
a central processing unit (CPU) 252, memories 253 and an interconnect bus 254. The CPU 252 
may contain a single microprocessor, or may contain a plurality of microprocessors for 
configuring the computer system 252 as a multi-processor system. The memories 253 include a 
main memory, a read only memory, and mass storage devices such as various disk drives, tape 
drives, etc. The main memory typically includes dynamic random access memory (DRAM) and 
high-speed cache memory. In operation, the main memory stores at least portions of data and of 
instructions for execution by the CPU 252. 

[0044] The mass storage may include one or more magnetic disk or tape drives or optical 

disk drives, for storing data and instructions for use by CPU 252. At least one mass storage 
system 255, preferably in the form of a disk drive or tape drive, stores the data and programming 
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related to the video dissemination services. The mass storage 255 may also include one or more 
drives for various portable media, such as a floppy disk, a compact disk read only memory (CD- 
ROM), or an integrated circuit non-volatile memory adapter (i.e. PCMCIA adapter) to input and 
output data and code to and from the computer system 251. 

[0045] The system 251 also includes one or more input/output interfaces for 

communications, shown by way of example as an interface 259 for data communications. The 
interface 259 may be a modem, an Ethernet card or any other appropriate data communications 
device. To perform as one of the servers 3, for the video dissemination to a large number of end 
user clients 5, the interface 259 preferably provides a relatively high-speed link to the network, 
although the final links to the individual clients 5 may use relatively narrow bandwidth 
technologies. The same or a separate communication interface also enables the server to receive 
telemetry and video data from one or more source systems 1 mounted on one or more UAVs or 
the like. If the video surveillance platform sends analog video, the server may have a separate 
receiver and video processing card to receive and digitize the video signal and parse the 
telemetry data. 

[0046] Although not shown, the system 251 may further include appropriate input/output 

ports for interconnection with a local display and a keyboard or the like serving as a local user 
interface for programming purposes. Alternatively, the server operations personnel may interact 
with the system 25 1 for control and programming of the system from a device similar to the 
system shown in Fig. 3, which may be locally connected thereto or connected to the server 3 via 
the Internet or some other network link. 

[0047] The computer system 251 runs a variety of applications programs and stores 

relevant data, such as the above noted programs for the near real-time video dissemination 
operations. For instance, in this example, the video processing function 7 is performed by the 
CPU 252 under software control. Those skilled in the art will recognize that all or portions of 
the video processing could be performed in hardware, for example, on a video processing card 
(not shown). Those skilled in the art will recognize that the computer system 251 may run other 
programs and/or host other applications. Also, each system 251 may be implemented as a single 
computer system or as a distributed system having multiple appearances at different nodes on a 
network. 
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[0048] The components contained in the computer systems 151 and 251 are those 

typically found in general purpose computer systems used as servers, workstations, personal 
computers, network terminals, and the like. In fact, these components are intended to represent a 
broad category of such computer components that are well known in the art. 
[0049] It may be helpful now to consider the main steps of operations of the NRTV video 

dissemination system. Figs. 5 to 7 are useful in understanding these operations. 
[0050] As shown in Fig. 5, a surveillance video source, from a manned platform or 

unmanned a erial v ehicle ( UAV), i s r eceived v ia a n etwork o r R F c ommunications 1 ink a s a n 
NTSC OR PAL analog signal or as a digital stream. The data will be decrypted by the receiving 
system, if it is encrypted. The streaming analog video will be fed into the video processor sub- 
component of the NRTV server. 

[0051] If not already digital, the incoming video is digitally encoded, time stamped 

(indexed) and converted into an MPEG-2 formatted video file via the video capture subsystem 
that is part of the video processor, and stored on local hard drives for later access. The local hard 
drives can nominally store an entire mission, which could last up to 8 hours or more. Increased 
storage capacity can be added. 

[0052] In parallel, the telemetry metadata from the surveillance sensor is parsed, 

validated, time stamped (indexed) and stored in a local hard drive. This processing of the 
telemetry data generates an ASCII file that contains sufficient parametric information 
(geographic location and technical sensor pointing information) necessary for the client display 
subsystem to graphically represent the sensor field of view data as overlays on a digital map. In 
the example, the overlays appear as four-sided polygons, typically rectangles. The ASCII data 
file is passed to the communications processor, for relaying to the client subsystem(s) over 
available communications (Network or RF communications system) (file transfer). 
[0053] In parallel with the ASCII file generation, video frames are sampled, time 

stamped (indexed) and stored locally by the VP server subsystem at the same rate that the field 
of view data is generated. The client device receives the formatted metadata from the 
communications processor. The client device parses the data into separate records, since the 
communications processor may send partial or multiple records in one transmission. The client 
device copies the data contained in each complete record into an internal structure. The client 
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device stores each complete data record into an internal SQL database. This allows archival 
retrieval and playback of the data. 

[0054] Using the location and field-of view sensor data, the geographic coordinates of 

the area on the earth that was seen by the image sensor for the specific instance of metadata can 
be calculated. The client device calculates these coordinates and creates a rectangular graphical 
object ('overlay'). The client device generates screen coordinates for each geographic coordinate 
and draws the overlay for each record. Associated with each overlay is the data received for 
each metadata record. This data is available to the user by use of the GUI associated with each 
overlay on the screen. 

[0055] With reference to Fig. 6, each overlay on the users' screen is associated, by time, 

with a single frame from the video stream. Using the client GUI, the user can select 1 or more 
overlays to request a series of video stills from the VP. The overlays selected need not be 
contiguous. The client will build a Tactical Information Request (TIR) with the time values 
associated with each selected overlay to be sent to the VP via the communications processor. 
[0056] At the server, the communications processor will pass each time value to the 

video processor. When the still is received from the video processor, the communications 
processor will send it to the client. T he c lient w ill receive the still and update the associated 
overlay to indicate graphically to the user that the still has been received. The user can select the 
still for display by itself or have the still overlaid on the current map. 

[0057] Using the client GUI, the user can select 2 overlays (a start of video and an end of 

video frame) to request a video clip for extraction from the video stream by the video processor. 
The client will build a TER with the time values associated with the 2 selected overlays to be sent 
to the video processor via the communications processor. The format for the video clip will be 
MPEG2, which is currently in general use. Future formats could include MPEG4, or other 
accepted video encoding standards. 

[0058] The communications processor validates the request from the client, for example, 

by authenticating the user and/or client device and confirming that the requested information is 
available (and/or is available to this user). The communications processor will build a request 
for the video processor containing the start time and stop time of the requested clip. When the 
clip is received from the VP, the communications processor will send it to the client, as shown in 
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Fig. 7. When the clip is received by the client device, the default video player software will play 
the clip automatically on the display of the user interface. 

[0059] Those skilled in the art will recognize that the concepts outlined above may be 

implemented in software programming, for example, as executed on the server 3 and the client 
devices 5. As such, the concepts herein encompass the programmed components as well as 
products bearing the relevant software. In alternative examples, hard-wired circuitry may be 
used in place of or in combination with software instructions to implement the NRTV 
functionalities. Thus, examples of the invention are not limited to any specific combination of 
hardware circuitry and software. 

[0060] A software product includes information that may be carried by at least one 

machine-readable medium. The information carried by the medium may be executable code, one 
or more databases, information regarding surveillance video available through the NRTV 
dissemination technology and/or actual video. 

[0061] Terms such as computer or machine "readable medium" as used herein refer to 

any medium that participates in providing instructions to a processor for execution. Such a 
medium may take many forms, including but not limited to, non-volatile media, volatile media, 
and t ransmission m edia. N on- volatile m edia i nclude, f or e xample, o ptical o r m agnetic d isks, 
such as storage devices 155 and 255. Volatile media include dynamic memory, such as main 
memory 153 and 253. Transmission media include coaxial cables; copper wire and fiber optics, 
including the wires that comprise bus 154 or bus 254. Transmission media can also take the 
form of electric or electromagnetic signals, or acoustic or light waves, such as those transported 
via wire or fiber optics or those generated during radio frequency (RF) and infrared (IR) data 
communications. Common forms of computer-readable media include, for example, a floppy 
disk, a flexible disk, hard disk, magnetic tape, any other magnetic medium, a CD-ROM, DVD, 
any other optical medium, punch cards, paper tape, any other physical medium with patterns of 
holes, a RAM, a PROM, and EPROM, a FLASH-EPROM, any other memory chip or cartridge, 
a carrier wave transporting data or instructions, a physical link transporting such a carrier wave 
or any other medium from which a computer or the like can read information. Various forms of 
computer readable media may be involved in carrying one or more sequences of one or more 
instructions to a system 151 or 251 or within the system to a processor 152 or 252 for loading 
into the computer system and/or for execution by the respective processor. 
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[0062] While the foregoing has described what are considered to be the best mode(s) 

and/or other examples, it is understood that various modifications may be made therein and that 
the subject matter disclosed herein may be implemented in various forms and examples, and that 
they may be applied in numerous applications, only some of which have been described herein. 
It is intended by the following claims to claim any and all modifications and variations that fall 
within the true scope of the present concepts. 



